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ABSTRACT 

The relationship between the importance of the 
objectives of a public school reading program and the extent of data 
collection efforts pertaining to these objectives was examined. The 
purpose of the study was to determine if the information collected 
was sufficient for teachers to make a continuous evaluation of the 
program's ability to meet the objectives. An instrument containing 
102 reading objectives, the Reading Objective Information Inventory 
(ROII) , was administered to 24 elementary teachers in 11 schools. 
Sixty objectives in the Roll were delineated as "important", and the 
availability of information to measure the achievement of these 60 
objectives was assessed. Results showed that (1) the relationship 
between the importance of the objectives and the overall availability 
of information to measure their achievement correlated .55; (2) in 
evaluating the effectiveness of data collection for each of the 
objectives, only nine objectives reflected a significant relationship 
between importance and data availability; (3) in comparing the 
differences in data collection effectiveness among three kinds of 
objectives (word recognition, comprehension, and work study skills) , 
the proportion of comprehension objectives with a significant 
importance-availability relationship was significantly higher than 
for word recognition or work study skills objectives. (VJ) 
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Statement of Problem 

The data collection aspect of evaluation has been identified as a 
most crucial element in effective evaluation. Moreover, efforts to 
improve data collection systems must be developed within the context of 
the present data collection system. The primary purpose/ f pf this study 
was to determine the extent to which the data collection efforts in a 
large, urban, Midwestern public school system's Title I reading program 
were providing the information required for classroom teachers to make 
a continuous evaluation of the program's ability to meet instructional 
objectives. 

The study was designed to provide an outline of the important objec- 
tives of the Title I reading program and an enumeration of those objectives 
for which information required to measure their achievement was being met 
through the present evaluation plan. The findings of the study could 
serve as the basis upon which recommendations for an improved data collec- 
tion system could be made. 
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Robbins, page 2 

Also, the techniques developed and validated for the purpose of 
identifying and classifying information requirements from program objec- 
tives should be applicable to programs other than reading. Accordingly, 
the techniques developed could be used in other program settings where 
there is an interest in analyzing the effectiveness of current evalua- 
tion efforts to meet the information requirements of classroom teachers 
to make effective decisions about reading instruction. 

Design of the Study 

The design of this study follows an evaluation paradigm rather than 
the more common experimental research designs. 

The evaluation design developed for this study was based upon the 
decision-centered model of Guba and Stuf flebeam (2) . The essential 
aspects of that model which were adopted for this study were: 

(1) focusing the evaluation to identify the questions to be 

answered and the criteria to be employed in answering them, (2) 

collecting information, (3) organizing information, (4) analyzing 

information, and (5) reporting information (p. 15). 

Consistent with that model, the design of this study included the 
process of posing questions and collecting information relative to the 
effectiveness of data collection efforts in the selected school district, 
organizing the information collected and establishing criteria for an- 
swering the questions posed, and posing hypotheses and identifying data 
treatment strategies for analyzing the information collected. 

The questions posed in the study were: 

1. What are the important objectives of the reading program in the 
selected school system? 

2. What is the overall relationship between the relative importance 
of reading program objectives and the degree to which information 
is available to measure the achievement of those objectives? 

3. What is the relationship between the relative importance of an 
objective and the degree to which information is available to 
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measure its achievement for each of the important reading program 
objectives? 

4. To what extent does the effectiveness of the data collection 
efforts in the selected school district vary among the three 
major aspects of reading objectives as identified in the B0IX, 
ie. word recognition, comprehension, and work study skills? 

The statistical analysis of the information collected to answer the 
questions posed in the study demanded that those questions be stated in 
the form of testable hypotheses. Statistical techniques were applied to 
questions 2, 3, and 4 of the study. The hypotheses required to test the 
information relative to those questions were: 

H 1. There is no significant correlation between the importance 
° of reading program objectives and the availability of in- 

formation to measure the achievent of those objectives. 

H 2. There is no significant relationship between the importance 
° of an objective of the reading program and the availability 

of information to measure its achievement. 

H 3. There are no significant differences among the major aspects 

° of reading, i. e. word recognition, comprehension, and work 

study skills in the proportions of objectives that achieved 

a significant relationship between their importance and the 

availability of information to measure their achievement 

under H 2. 
o 

Procedures of the Study 

Twenty-four subjects were selected from among volunteers to partici- 
pate in the study. These subjects represented four classroom teachers 
from each grade, one through six, and were drawn from 11 different Title 
1 target schools in the school district in which the study was conducted. 

The answer to the questions posed for this study required the collec- 
tion of information relative to both the importance of the objectives of 
the reading programs in the classrooms of the selected subjects and the 
degrees to which information to measure the acheivement of those objectives 
was collected and available. Because the literature suggested no instru- 
ment and/or techniques were available to collect such information, the 
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investigator developed such an instrument. 

That instrument, the Reading Objective Information Inventory, (R0II) 
was developed by outlining the skills covered by the reading objectives 
included in the Instructional Objectives Exchange Catalogs (1) for grades 
kindergarten through six, and then selecting one objective for each of 
the skills included in the outline. The objectives were either selected 
verbatum or were recast from one or more of the objectives included in 
the Instructional Objectives Exchange Catalogs covering the skills iden- 
tified in the outline. This process resulted in 102 reading objectives 
being selected for inclusion in the ROII. 

After selecting the objectives to be rated, scales were developed 
for rating both the importance of the objectives and the availability of 
information to measure their achievement. The ROII was then administered 
to the 24 subjects in order to collect information for the study to assess 
the effectiveness with which data to measure the objectives of the. read- 
ing program were being collected. 

Findings of the Study 

The first requirement of the study design was the delineation of the 
"important" objectives of the reading program. This was accomplished by 
identifying those objectives cu the ROII that achieved a mean importance 
rating of 1.5 or higher, the arbitrary criterion level established in the 
study for being considered important. Of the 102 objectives listed in 
the ROII , 60 achieved criterion level. These 60 objectives were thus 
identified as the objectives of the reading program for which the availa- 
bility of information to measure their achievement would be assessed. 

The first approach to assessing the effectiveness of data collection 
efforts was to consider the overall or general effectiveness of data 
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collection efforts in the selected school district. This was accomplished 
by measuring the relationship between the importance of reading program 
objectives and the availability of information to measure the achieve- 
ment of those objectives. A Pearson product moment correlation coeffi- 
cient of .55 was found between the two variables. 

The second approach was to evaluate the effectiveness of data collec- 
tion efforts for each of the 60 important reading objectives. This was 
accomplished by measuring the relationship between the importance of an 
objective and the av&ilability of information to test its accomplishment. 
Two methods were used to identify those objectives having such a signifi- 
cant relationship. A Fisher exact probabilities test was computed to 
test the relationship between the importance of an objective and the 
availability of information to assess its achievement for each objective. 
Six objectives reflected a significant relationship through Che test. 

The second method was' to identify those objectives that achieved a mean 
availability rating (the degree to which information to measure their 
achievement was collected and available) that equaled or exceeded the ar- 
bitrary criterion level of 1.5 established for the study. Six objectives 
were identified with availability ratings reaching criterion level or 
higher. 

A comparison of the 12 objectives identified through the two methods 
revealed that three of the 12 objectives identified through the two me- 
thods were common for both methods. The result, therefore, was that a 
total of nine different reading objectives reflected a significant rela- 
tionship between their importance and the availability of information to 
measure their achievement. 

The third approach to evaluating the effectiveness of data collection 
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efforts in the selected school system was to compare the differences in 
data collection effectiveness among three different aspects of reading 
program objectives, i.e. word recognition, comprehension, and work study 
skills. This was accomplished by determining whether or not a significantly 
different proportion of the objectives in each of these areas reflected 
significant importance-availability relationships. Of the nine objectives 
that reflected such relationships, three were word recognition objectives 
and six were comprehension objectives, paring the proportions that theae 
figures reflected, i.e. three of 32 word recognition objectives, or. .09; 
six of 14 comprehension objectives, or .49; and zero of 14 work study 
objectives, or .00; it was found that the proportions of comprehension 
objectives with significant importance-availability relationship was sig- 
nificantly higher than the proportion of such objectives for either word 
recognition or work study skills. 

Conclusions of the S tudy 

From the findings of the study it can be concluded that although 
there was a relationship between the importance of reading objectives 
in the selected school district and the degree to which information 
was available to measure the achievement of those objectives as 
reflected by the correlation coefficient of .55 that relationship 
was well below that which could reasonably be considered adequate. 
Accordingly, it can be stated that as a general conclusion of this 
study, efforts in the selected school system to collect data relative 
to the measurement of the achievement of the objectives of the Title 
I reading program in selected classrooms were inadequate. 
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This study found that adequate information was collected for measur- 
ing the achievement of only nine of the 60 important reading objectives. 

From these findings it can be concluded that a data collection effort which 
provides information to measure the achievement of only 15 per cent of the 
important objectives of a reading program, when a reasonable goal would be 
the collection of adequate information to assess the achievement of all 
objectives, is certainly inadequate. 

From the findings of the study that information was available to measure 
the achievement of a significantly higher proportion of objectives in the 
area of comprehension than in either word recognition or work study, it can 
be concluded that data collection was more effective in providing information 
to measure the achievement of comprehension objectives than in providing 
information to measure the accomplishment of either word recognition or 
work study objectives. 
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